An introduction to stochastic approximation

We propose to start the exposition of the topic by an example. The arguments are given in a crude manner. Formal proofs will be given in section 2. This example is taken from the very article [6] which introduced stochastic approximation. Consider x ∈ R the parameter of a system and g(x) ∈ R an output value from this system when parameter x is used. We assume g to be a smooth, increasing function. An agent wants to determine sequentially x∗ ∈ R the value such that the system output equals a target value g∗. If for all x, the value of g(x) can be observed directly from the system, then determining g∗ could be solved by a simple search technique such as binary search or golden ratio search. Here we assume that only a noisy version of g can be observed. Namely, at time n ∈ N, the decision maker sets the parameter equal to xn, and observes Yn = g(xn) +Mn with Mn a random variable denoting noise, with E[Mn] = 0. In order to determine g(x), a crude approach would be to sample parameter x repeatedly and average the result, so that the effect of noise would cancel out, and apply a deterministic line search (such as binary search). [6] proposed a much more elegant approach. If xn > x ∗ , we have that g(xn) > g ∗, so that diminishing xn by a small amount proportional to g −g(xn) would guarantee xn+1 ∈ [x∗, xn]. Therefore, define n a sequence of small positive numbers, and consider the following update scheme: xn+1 = xn + n(g ∗ − Yn) = xn + n(g − g(xn)) + nMn.